Papers with scaling Large Language Models
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages (2023.acl-long)
Copied to clipboard
Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, Hinrich Schütze
| Challenge: | Lack of LLMs supporting low-resource languages is a serious impediment to bringing NLP to all of the world. |
| Approach: | They create a model that scales LLMs horizontally and a corpus that covers 511 low-resource languages. |
| Outcome: | The proposed model improves on five diverse tasks across low- and high-resource languages. |
COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distribution (2026.findings-acl)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) architectures are effective for scaling Large Language Models (LLMs) however, existing pruning methods adopt uniform pruning across layers, which fails to capture layer-wise variations in expert importance and redundancy. |
| Approach: | They propose a Mixture-of-Experts pruning method that activates only a subset of experts during inference by estimating expert importance using Fisher information. |
| Outcome: | The proposed pruning method outperforms existing pruning methods while reducing inference latency and peak GPU memory usage. |